Skip to content

[schema-descriptions] Carry Choice and Score question text inside the schema definition - #31

Merged
egafni merged 3 commits into
mainfrom
fix/choice-score-descriptions-survive-ref
Sep 6, 2026
Merged

egafni merged 3 commits into
mainfrom
fix/choice-score-descriptions-survive-ref

Conversation

@hackgoofer

@hackgoofer hackgoofer commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Choice and Score answers are nested probability-map models, which the JSON schema emits as a $ref with the question and its option criteria in a description beside the reference. pydantic-ai's OpenAI transformer inlines the $ref and merges that description; its Anthropic transformer keeps $defs and the description beside the $ref does not reach the model. Captured requests to Anthropic showed intent as a bare $ref whose definition listed eight option keys described only as {maximum: 1, minimum: 0}: the model saw no question and no criteria, and answered from the key names.

Set the description as the probability-map model's docstring as well, so pydantic emits it as the definition's own description, inside $defs, where it survives either treatment of the reference. Verified on a captured request to claude-opus-5: the question text is present, and a refund case the model read as delivery_problem 0.60 with the text missing reads as refund_request 0.82 with it present, matching a direct API call.

Before the fix (main). The intent property is a bare reference, and the definition it points to has no description:

  "intent": {"$ref": "#/$defs/ProbabilityMap0"}

  "ProbabilityMap0": {
    "type": "object",
    "additionalProperties": false,
    "required": ["unauthorized_charge", "card_not_working", "refund_request", "billing_error",
                 "delivery_problem", "account_access", "cancel_account", "general_inquiry"],
    "properties": {
      "refund_request": {"type": "number", "description": "{maximum: 1, minimum: 0}"},
      ...
    }
  }

The model's answer: refund_request 0.49, delivery_problem 0.49. A coin toss between the two names.

After the fix (fix/choice-score-descriptions-survive-ref). The property is the same bare reference, but the definition now carries the question and every option's criteria:

  "intent": {"$ref": "#/$defs/ProbabilityMap0"}

  "ProbabilityMap0": {
    "type": "object",
    "description": "Each property maps an option to the probability that it is the best answer.
      Question: What does the customer want above anything else in this conversation?
      Required probability keys:
      unauthorized_charge = The customer wants a charge, transfer, or login they say they did not make investigated and removed, and the account protected.
      card_not_working = The customer wants their own card to work again: it was declined, blocked, or is otherwise unusable.
      refund_request = The customer wants money back above anything else, for something they bought, whatever happened to it: it never arrived, arrived damaged, was wrong, or they changed their mind. The remedy asked for is the money.
      billing_error = ...
      delivery_problem = The customer wants the item above anything else: delivered, found, re-sent, or replaced. Money back is not the remedy they are after.
      ...",
    "additionalProperties": false,
    "required": [...],
    "properties": { "refund_request": {"type": "number", "description": "{maximum: 1, minimum: 0}"}, ... }
  }

hackgoofer and others added 3 commits September 6, 2026 09:07
Choice and Score answers are nested probability-map models, which the JSON
schema emits as a $ref with the question and its option criteria in a
description beside the reference. pydantic-ai's OpenAI transformer inlines
the $ref and merges that description; its Anthropic transformer keeps $defs
and the description beside the $ref does not reach the model. Captured
requests to Anthropic showed intent as a bare $ref whose definition listed
eight option keys described only as {maximum: 1, minimum: 0}: the model
saw no question and no criteria, and answered from the key names.

Set the description as the probability-map model's docstring as well, so
pydantic emits it as the definition's own description, inside $defs, where
it survives either treatment of the reference. Verified on a captured
request to claude-opus-5: the question text is present, and a refund case
the model read as delivery_problem 0.60 with the text missing reads as
refund_request 0.82 with it present, matching a direct API call.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@egafni egafni changed the title Carry Choice and Score question text inside the schema definition [schema-descriptions] Carry Choice and Score question text inside the schema definition Sep 6, 2026
@egafni
egafni merged commit 27f2315 into main Sep 6, 2026
@danielgafni
danielgafni deleted the fix/choice-score-descriptions-survive-ref branch September 15, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants